Hamilton-Jacobi-Bellman Equation Research Articles

Relaxation and asymptotic expansion of controlled stiff differential equations

The control of relaxation-type systems of ordinary differential equations is investigated using the Hamilton–Jacobi–Bellman equation. Firstly, we recast the model as a singularly perturbed dynamics which we embed in a family of controlled systems. Then we study this dynamics together with the value function of the associated optimal control problem. We provide an asymptotic expansion in the relaxation parameter of the value function. We also show that its solution converges toward the solution of a Hamilton–Jacobi–Bellman equation for a reduced control problem. Such systems are motivated by semi-discretisation of kinetic and hyperbolic partial differential equations. Several examples are presented including Jin–Xin relaxation.

Optimal control of stochastic differential equations with random impulses and the Hamilton–Jacobi–Bellman equation

AbstractIn this article, we study the optimal control of stochastic differential equations with random impulses. We optimize the performance index and add the influence of random impulses to the performance index with a random compensation function. Using the idea of stochastic analysis and dynamic programming principle, a new Hamilton–Jacobi–Bellman (HJB) equation is obtained, and the existence and uniqueness of its viscosity solution are proved.

Reliable optimal controls for SEIR models in epidemiology

We present and compare two different optimal control approaches applied to SEIR models in epidemiology, which allow us to obtain some policies for controlling the spread of an epidemic. The first approach uses Dynamic Programming to characterise the value function of the problem as the solution of a partial differential equation, the Hamilton–Jacobi–Bellman equation, and derive the optimal policy in feedback form. The second is based on Pontryagin’s maximum principle and directly gives open-loop controls, via the solution of an optimality system of ordinary differential equations. This method, however, may not converge to the optimal solution. We propose a combination of the two methods in order to obtain high-quality and reliable solutions. Several simulations are presented and discussed, also checking first and second order necessary optimality conditions for the corresponding numerical solutions.

Scalable computation of energy functions for nonlinear balanced truncation

Nonlinear balanced truncation is a model order reduction technique that reduces the dimension of nonlinear systems in a manner that accounts for either open- or closed-loop observability and controllability aspects of the system. A computational challenges that has so far prevented its deployment on large-scale systems is that the energy functions required for characterization of controllability and observability are solutions of various high-dimensional Hamilton–Jacobi–(Bellman) equations, which are computationally intractable in high dimensions. This work proposes a unifying and scalable approach to this challenge by considering a Taylor-series-based approximation to solve a class of parametrized Hamilton–Jacobi–Bellman equations that are at the core of nonlinear balancing. The value of a formulation parameter provides either open-loop balancing or a variety of closed-loop balancing options. To solve for the coefficients of Taylor-series approximations to the energy functions, the presented method derives a linear tensor system and heavily utilizes it to numerically solve structured linear systems with billions of unknowns. The strength and scalability of the algorithm is demonstrated on two semi-discretized partial differential equations, namely the Burgers and the Kuramoto–Sivashinsky equations.

An analytical adaptive optimal control approach without solving HJB equation for nonlinear systems with input constraints

An analytical adaptive optimal control approach without solving HJB equation for nonlinear systems with input constraints

A comparison principle for semilinear Hamilton–Jacobi–Bellman equations in the Wasserstein space

A comparison principle for semilinear Hamilton–Jacobi–Bellman equations in the Wasserstein space

Optimization for stochastic model arisen from investment problem associated with default risk

In this paper, we study the optimal strategy arisen from the excess-of-loss reinsurance and asset allocation in defaultable markets under a general stochastic model. By developing the corresponding dynamic programming approach, we establish the optimal investment approach through two sub-problems: a pre-default case and a post-default case, respectively, characterized by the obtained Hamilton–Jacobi–Bellman (HJB) equations. We show the existence of a classical solution to the pre-default case via super-sub solution techniques and give an explicit characterization of the optimal reinsurance and investment policy that maximizes the common used utility associated with the terminal wealth. Verification theorem is further established to show the uniqueness of the corresponding solution of HJB equation that is critical to the desired optimal solution.

Minimum loss optimization of flywheel energy storage systems via distributed adaptive dynamic programming

AbstractIn this article, a distributed controller based on adaptive dynamic programming is proposed to solve the minimum loss problem of flywheel energy storage systems (FESS). We first formulate a performance function aiming to reduce total losses of FESS in power distribution applications. Then we use the Hamilton–Jacobi–Bellman (HJB) equation to solve this optimal control problem. The solution of the HJB equation is approximated by neural networks. To achieve distributed control, we estimate the global variables in the HJB equation by using the dynamic average consensus algorithm. A barrier Lyapunov function and a saturation function are introduced to handle the issue of state and input constraints, respectively. Then the stability of the system is proved through the Lyapunov stability analysis. Finally the effectiveness of the proposed strategy is verified by simulations. Simulation results show that FESS can track the power command while minimizing total power losses by interacting with neighbors. The proposed algorithm leads to a loss reduction of compared to the equal power distribution strategy.

Viscosity solutions approach to finite-horizon continuous-time Markov decision process

This paper investigates the optimal control problems for the finite-horizon continuous-time Markov decision processes with delay-dependent control policies. We develop compactification methods in decision processes and show that the existence of optimal policies. Subsequently, through the dynamic programming principle of the delay-dependent control policies, the differential-difference Hamilton-Jacobi-Bellman (HJB) equation in the setting of discrete space is established. Under certain conditions, we give the comparison principle and further prove that the value function is the unique viscosity solution to this HJB equation. Based on this, we show that among the class of delay-dependent control policies, there is an optimal one which is Markovian.

Cobra aerobatic flight for quadrotors using θ-D nonlinear control

The control design for the Cobra maneuver of quadrotors is presented based on quaternion dynamics. Cobra maneuver commands the quadrotor in ± π / 2 (rad) pitch angle such that the conventional hovering assumption for quadrotor flight is not applicable. The finite-time θ-D method is proposed to control the Cobra maneuver. Based on this technique, the intractable Hamilton–Jacobi–Bellman equation can be approximated by a differential Riccati equation and a series of state-dependent Lyapunov equations, which can be solved recursively and lead to the closed-form state feedback control law . This control law can achieve semi-global asymptomatic stability of the closed-loop system. Numerical simulations verify that the proposed technique is compared favourably with the finite-time state-dependent Riccati equation (SDRE) technique in terms of computation efficiency and control effort, and also shows faster recovery of the quadrotor after the Cobra maneuver compared with conventional dynamics without the derivative of the rotation matrix.

Data-Driven Stochastic Optimal Control With Safety Constraints Using Linear Transfer Operators

We provide a data-driven framework for optimal control of a continuous-time control-affine stochastic dynamical system. The proposed framework relies on the linear operator theory involving Perron-Frobenius (P-F) and Koopman operators. Our first result involving the P-F operator provides a convex formulation to the optimal control problem in the dual space of densities. This convex formulation of the stochastic optimal control problem leads to an infinite-dimensional convex program. The finite-dimensional approximation of the convex program is obtained using a data-driven approximation of the linear operators. We provide rigorous error bounds and convergence rate for the data-driven approximation of the linear operators and comment on the convergence rate of the optimal control w.r.t. data size. Our second results demonstrate using the Koopman operator, dual to the P-F operator, for the stochastic optimal control design. We show that the Hamilton Jacobi Bellman (HJB) equation can be expressed using the Koopman operator. We provide an iterative procedure along the lines of a popular policy iteration algorithm based on the data-driven approximation of the Koopman operator for solving the HJB equation. The two formulations, namely the convex formulation involving the P-F operator and the Koopman-based formulation using the HJB equation, can be viewed as dual to each other where the duality follows due to the dual nature of the P-F and Koopman operators. Finally, we present examples to demonstrate the efficacy of the developed framework and verify the convergence rates for the operator and optimal control numerically.

Adaptive Virotherapy Strategy for Organism With Constrained Input Using Medicine Dosage Regulation Mechanism.

In this article, the constrained adaptive control strategy based on virotherapy is investigated for organism using the medicine dosage regulation mechanism (MDRM). First, the tumor-virus-immune interaction dynamics is established to model the relations among the tumor cells (TCs), virus particles, and the immune response. The adaptive dynamic programming (ADP) method is extended to approximately obtain the optimal strategy for the interaction system to reduce the populations of TCs. Due to the consideration of asymmetric control constraints, the nonquadratic functions are proposed to formulate the value function such that the corresponding Hamilton-Jacobi-Bellman equation (HJBE) is derived which can be deemed as the cornerstone of ADP algorithms. Then, the ADP method of a single-critic network architecture which integrates MDRM is proposed to obtain the approximate solutions of HJBE and eventually derive the optimal strategy. The design of MDRM makes it possible for the dosage of the agentia containing oncolytic virus particles to be regulated timely and necessarily. Furthermore, the uniform ultimate boundedness of the system states and critic weight estimation errors is validated by Lyapunov stability analysis. Finally, simulation results are given to show the effectiveness of the derived therapeutic strategy.

A numerical method for ergodic optimal control of switching diffusions with reflection

The ergodic or long run average cost control problem for diffusions is one of the classical problems of stochastic control that still eludes a completely satisfactory treatment. This is certainly true for the setting in which the systems to be controlled is modeled by the solution of a switching stochastic differential equation with reflection (SSDER). In this paper, we advance in this rather difficult problem by setting forth preliminary results for the unidimensional case. Besides carving out a numerical method, our treatment of the ergodic control in this scenario straddles issues of existence and uniqueness of solution of the SSDER and a verification theorem for the associated HJB equation. We conclude by illustrating the effectiveness of the method considering the control of energy consumption in a large parallel processing computer system composed of one queue and several processing stations.

Markov Chain Approximation for Hamilton–Jacobi–Bellman Equation with Absorbing Boundary

Markov Chain Approximation for Hamilton–Jacobi–Bellman Equation with Absorbing Boundary

Optimized leader–follower consensus control for high-order nonlinear multi-agent system modeled in canonical dynamic form

This paper addresses the optimized leader–follower consensus control problem of nonlinear canonical dynamic multi-agent system (MAS). Since every agent is modeled in high-order dynamic, they contain the various state variables having derivative relations. Consequently, to attain the optimized consensus control, backstepping technique and reinforcement learning (RL) are combined together. In the first backstepping step, the virtual control is formulated to have the consensus error term consisting of neighbor agents’ output states. In the last backstepping step, since the nonlinearity dynamic is arisen, the optimized actual control is derived by performing the critic-actor RL. It is worth mentioning that, in the traditional RL optimal control, the critic and actor updating laws are obtained by performing the gradient descent algorithm to square of the approximated Hamilton–Jacobi-Bellman (HJB) equation, which includes multiple nonlinearity items, as a result, the algorithm is very intricate. However, in this optimized consensus control, since the RL training laws are yielded in the light of negative gradient of a simple positive function that is relevant to the HJB equation, its algorithm is very simple. At the same time, it can also avoid the persistence excitation conditions. Finally, theory and simulation confirm the validity of this optimized consensus method.

Robustness improvement of optimal control in terms of RBFNN with empirical model reduction and transfer learning

This paper proposes a method to compute solutions of optimal controls for dynamic systems in terms of radial basis function neural networks (RBFNN) with Gaussian neurons. The RBFNN is used to compute the value function from Hamilton–Jacobi–Bellman equation with the policy iteration. The concept of dominant system is introduced to create initial coefficients of the neural networks to stabilise unstable systems and guarantee the convergence of policy iteration. Model reduction and transfer learning techniques are introduced to improve robustness of RBFNN optimal control, and reduce computational time. Numerical and experimental results show that the resulting optimal control has excellent control performance in stabilisation and trajectory tracking, and much-improved robustness to disturbances and model uncertainties even when the system responses move to a domain of the state space that is larger than the domain where the neural networks are trained.

Optimal synchronization for multi-agent systems: A performance-dependent switching topology

This paper proposes an optimal output synchronization control method for heterogeneous multi-agent systems (HMASs) under a performance-dependent switching topology and DoS attacks. First, local and global switched performance index functions (SPIF) and SPIF-dependent topology switching law are proposed, respectively, thus, the control performance and topology quality can be quantitatively expressed. Second, an adaptive dynamic programming (ADP) algorithm with mode switching is proposed, aimed at dealing with the difficult Hamilton–Jacobi–Bellman equation, as well as the analytical complexity caused by the switching dynamics. The convergence of the switched ADP algorithm is proven to ensure its correct implementation. Then, for different topologies, multi-mode Actor–Critic neural networks (NNs) are built for each agent to calculate optimized control policies and SPIF, respectively. Furthermore, an NN-based state compensation mechanism is designed to expand the applicability of the designed switched ADP algorithm when the leader’s output transmission is unreliable. Finally, the results of numerical examples confirm that the proposed method is feasible.

Bellman Neural Networks for the Class of Optimal Control Problems With Integral Quadratic Cost

This work introduces Bellman Neural Networks (BeNNs) and employs them to learn the optimal control actions for the class of optimal control problems (OCPs) with integral quadratic cost. BeNNs represent a particular family of Physics-Informed Neural Networks (PINNs) specifically designed and trained to tackle OCPs via applying the Bellman Principle of Optimality (BPO). The BPO provides necessary and sufficient optimality conditions, which result in a nonlinear partial differential equation known as the Hamilton-Jacobi-Bellman (HJB) equation. BeNNs learn the optimal control actions from the unknown solution of the arising HJB equation (i.e., the value function), where the unknown solution is modeled using a Neural Network. Additionally, the paper shows how to estimate the upper bounds on the generalization error of BeNNs while learning the solutions for the OCP class under consideration. The generalization error estimate is provided in terms of the choice and number of the training points as well as the training error. Numerical studies show that BeNNs can be successfully applied to learn the feedback control actions for the class of optimal control problems considered and, after the training is completed, deployed to control the system in a closed-loop fashion. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Impact Statement</i> —The proposed research improves our understanding of how to solve optimal control problems with closed-loop solutions and has potentially a countless number of applications in several different areas. The study is at the intersection between optimal control theory and artificial intelligence connected with mathematical tools for functional interpolation. This advances the ability to implement a higher level of autonomy in decision-making for practical applications with a beneficial impact on our society.

Optimized tracking control using reinforcement learning and backstepping technique for canonical nonlinear unknown dynamic system

AbstractThe work addresses the optimized tracking control problem by combining both reinforcement learning (RL) and backstepping technique for the canonical nonlinear unknown dynamic system. Since such dynamic system contains multiple state variables with differential relation, the backstepping technique is considered by making a virtual control sequence in accordance with Lyapunov functions. In the last backstepping step, the optimized actual control is derived by performing the RL under identifier‐critic‐actor structure, where RL is to overcome the difficulty coming from solving Hamilton‐Jacobi‐Bellman (HJB) equation. Different from the traditional RL optimizing methods that find the RL updating laws from the square of the HJB equation's approximation, this optimized control is to find the RL training laws from the negative gradient of a simple positive definite function, which is equivalent to the HJB equation. The result shows that this optimized control can obviously alleviate the algorithm complexity. Meanwhile, it can remove the requirement of known dynamic as well. Finally, theory and simulation indicate the feasibility of this optimized control.

Nonlinear Optimal Control for Stochastic Dynamical Systems

This paper presents a comprehensive framework addressing optimal nonlinear analysis and feedback control synthesis for nonlinear stochastic dynamical systems. The focus lies on establishing connections between stochastic Lyapunov theory and stochastic Hamilton–Jacobi–Bellman theory within a unified perspective. We demonstrate that the closed-loop nonlinear system’s asymptotic stability in probability is ensured through a Lyapunov function, identified as the solution to the steady-state form of the stochastic Hamilton–Jacobi–Bellman equation. This dual assurance guarantees both stochastic stability and optimality. Additionally, optimal feedback controllers for affine nonlinear systems are developed using an inverse optimality framework tailored to the stochastic stabilization problem. Furthermore, the paper derives stability margins for optimal and inverse optimal stochastic feedback regulators. Gain, sector, and disk margin guarantees are established for nonlinear stochastic dynamical systems controlled by nonlinear optimal and inverse optimal Hamilton–Jacobi–Bellman controllers.